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Topological band insulators which are dynamically generated by electron-electron interactions have been the- 
oretically proposed in two and three dimensional lattice models. We present evidence that the two-dimensional 
version can be stabilized in digital (111) heterostructures of transition-metal oxides as a result of purely local 
interactions. The topological phases are accompanied by spontaneous ordering of complex orbitals and we 
discuss their stability with respect to the Hund's rule coupling, Jahn-Teller interaction and inversion symmetry 
breaking terms. As main competitors we identify spin-nematic and magnetic phases. 



I. INTRODUCTION 

The search for new materials which realize a topological 
insulator (TI) phase ^""^ has dramatically increased, recently. 
Thereby, the main focus has been on compounds involving 
heavy elements with strong spin-orbit coupling. ^'^ On the 
other hand, it was pointed out that topological band proper- 
ties can also arise from the spontaneous breaking of a sym- 
metry in interacting systems where the spin-orbit coupling 
is negligible.^ In principle, this scenario suggests that TIs 
can be found in a much larger class of materials and sev- 
eral theoretical investigations support the existence of such 
interaction-driven TIs in two and three-dimensional interact- 
ing lattice models. From the experimental point-of-view, 
the situation is less satisfying and an experimental study of 
an interaction-driven TI is still lacking. In fact, the num- 
ber of possible experimental systems is rather limited, and 
the most promising candidate so far is probably few-layer 
graphene.^^'^^ In this work, we build on previous theoretical 
investigations and show that a two-dimensional interaction- 
driven TI phase may be stabilized from purely local interac- 
tions in multi-orbital models for transition-metal oxides. This 
result significantly extends the range of possible experimen- 
tal systems. Using the conventional Hartree-Fock mean-field 
theory in combination with the theoretical analysis of the k p- 
model we find that the TI phase is accompanied by the spon- 
taneous ordering of complex orbitals. 

Our starting point is a system which belongs to the recently 
proposed class of digital oxide heterostructures ^"^'^^ grown in 
the (111) direction^^ and it is sketched in Fig. 1. More pre- 
cisely, we focus on the d-electrons of a (111) bilayer of the 
cubic transition-metal oxide ABO3 [see Fig. 1(b)] which is 
embedded in a band insulator AB'Oa. We assume that the 
"active" compound (ABO3) is metallic in bulk with a low spin 

configuration of the transition-metal (TM) ions, i.e. we as- 
sume filled t2g orbitals and one electron in the Cg manifold. 
A possible choice of materials satisfying these requirements 
is a (111) bilayer of LaNiOs embedded in the band insulator 
LaAlOa. 

The orbital degrees of freedom of the Cg manifold are 
described by the two real orbitals \a) = \dz2) and \b) = 
\dx2_y2), see Fig. 1(c), which form a T = 1/2-pseudo-spin 

T. We will argue that topological phases can be stabilized 
by a spontaneous (and possibly spin dependent) ferro-orbital 




FIG. 1. (Color online.) (a) The digital oxide heterostructure consid- 
ered in this article is grown in the (111) direction and of the form 
AB'Os/ABOs/AB'Oa. (b) The "active" region consists of a (111) 
bilayer of the metallic ABO3 perovskite. Shown are the locations of 
the transition-metal ions (B). (c) The bilayer system forms a honey- 
comb lattice when projected to the plane perpendicular to (1 1 1). We 
assume that the relevant orbital degrees of freedom are the eg orbitals 
of the transition-metal ions. 

ordering of complex orbitals of the form 

\d ± id) = {\d,2) ± i\d,2_y2)) /V2. (1) 

These orbitals are eigenstates of T^. In most cases, the com- 
plex orbitals Eq. (1) are energetically disfavored because both 
super-exchange and lattice distortions prefer real orbitals in 
stoichiometric compounds. However, the band structure of 
the considered (111) bilayer features a quadratic band cross- 
ing (QBC) point with a (i-wave symmetry in orbital space 
which favors ordering of complex orbitals in a range of pa- 
rameters at weak interactions.^ 



II. BILAYER MODEL 

As illustrated in Fig. 1(b) and (c), the transition-metal ions 
of the (111) bilayer system form a honeycomb lattice. We 
study the following effective Hamiltonian for the Cg electrons 
hopping on this honeycomb lattice 

^ = ^0 + ^int + ^perp- (2) 

The band Hamiltonian Hq in the tight-binding approximation 
has been derived in Ref. [16]. In momentum space it takes the 
form 

7^o = ^4(fc)^fo(feK(fe) (3) 

k,a 

where d^ = ((iiaa, <^i6cr, <^2acr, d2ba)^ is a vector of fermionic 
annihilation operators. Here, the bottom layer is labeled with 
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the subscript 1 and the top layer with the subscript 2. The 
orbital labels are a and b and the spin is a. The Bloch matrix 
i7o(^) is a 4 X 4 matrix of the form 



(a)0.3 



Ho{k) 



/ Sak ^abk^ 
£abk Sbk 

<k <bk 



(4) 



Here, we kept only the dominant nearest-neighbor hop- 



ping t and Eak = -t[l + ^cos{Ykx)e '^^^J, e^k = 



- Y cos(^/Ca,)e"^^^^ andsabfc = -i^tsm{^kx)e~''^'^y . 
[kx and ky directions refer to the (X, F)-axes in Fig. 1(c).] 
A more general form which also includes the second neigh- 
bor hopping is reproduced in the supplemental materials. ^^'^^ 
Our main conclusions remain valid as long as t is large com- 
pared to other tight-binding parameters. The non-interacting 
band structure of Eq. (4) has an interesting and for the fol- 
lowing discussion crucial feature: the Fermi surface at quar- 
ter filling consists of a single Fermi point k = where two 
bands touch quadratically.^^'^^ This QBC point has a d-wave 
symmetry in orbital space and a six-fold rotation symmetry 
in /c-space which protects it from splitting into Dirac points.^ 
We note here that the angular momentum of the Cg manifold 
is quenched and spin-orbit coupling only enters as a higher- 
order process via coupling to the t2g orbitals^^ which is as- 
sumed to be weak and neglected in the following. Further- 
more, the linear coupling to the trigonal crystal field is also 
absent^^ and the QBC point in the non-interacting band struc- 
ture is a rather generic feature of the considered heterostruc- 
ture. 

The electron-electron interaction is accounted for by the lo- 
cal interaction between the d-electrons of the form 

^int= y^^rira^nral + {U^ - J) ^ TlraarirfSa 



r a 



a>(3,a 



(5) 



The intra-orbital repulsion is denoted by U, the inter-orbital 
interaction by U\ J parametrizes the Hund's rule coupling 
and / the pair-hopping term. We employ the standard relations 
U = U' -\-2J and J = I valid for an isolated ion which leaves 
us with two independent interaction parameters U and J. 



The ideal electronic model is perturbed by Hp 



l-LjT- Here, Hy describes a sublattice potential which breaks 
the inversion symmetry between top and bottom layer 

= E [dLAk)dlaAk) - dL,(fc)d2aa(fe)] . (6) 



This term is present if the bilayer system is capped by a dif- 
ferent insulator than the one beneath it. Finally, H jt accounts 
for the cooperative Jahn-Teller effect which potentially drives 
a structural phase transition with distorted oxygen octahedra. 
The coupling of the electrons to the phonons of the oxygen 
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FIG. 2. (a) Zero temperature mean-field phase diagram of the ideal 
model as function of the repulsion U/t and Hund coupling J/U. 
Topological phases are found in the small- [/ limit (QAH2/QSH) and 
within the ferromagnetic phase (FM+QAHi). (b) Mean-field phase 
diagram as function of an inversion- symmetry breaking field V/t and 
a Jahn-Teller interaction of strength K/t for fixed interaction param- 
eters [* in (a)]. We find a topological phase between a spin nematic 
(SNz) and an antiferromagnetic (AFM) phase. More details about 
the various phases are given in the main text. 



displacements leads to an effective interaction between the 
electrons of neighboring transition-metal ions and we adapt 
the simple form^^'^^ 



(7) 



The suffix / = x^y^z denotes the direction of the bond be- 
tween i and j and = cos(^)i;^ - sin(^)i;^ with 
{rix^riy^nz) = (1, 2, 3). i^T is positive and therefore favors a 
staggered orbital order of real orbitals. 



III. PHASE DIAGRAM 



We first focus on the ideal electronic model and assume 
'Wperp = 0. The mean-field phase diagram obtained by solv- 
ing the self-consistency equations numerically is shown in 
Fig. 2(a) as a function of the two dimensionless interaction 
parameters U/t and J/U. We will understand the small-/7 
phases in this diagram qualitatively by analyzing the instabil- 
ities of the QBC in the next section. The strongly interact- 
ing limit is dominated by magnetic phases: If the Hund cou- 
pling J/U is sufficiently small, we find an antiferromagnetic 
(AFM) phase which is accompanied by a ferro-orbital (FO) 
order. For larger ratios of J/U we find (fully polarized) ferro- 
magnetic (FM) order. In the absence of orbital order, the FM 
phase is gapless and has two Dirac nodes. Orbital order can 
open a gap in the FM phase. In particular, a quantum anoma- 
lous Hall state^ (QAHi) with Chem number n = ±1 is found 
if complex orbitals are involved. The topological nature of 
this phase can be understood in analogy to the small- situa- 
tion discussed below. 

The weak to intermediate interaction regime is dominated 
by phases which are characteristic of the underlying QBC 
point. For small ratios J/U, we find an interaction-driven 
topological phase. The topological state either breaks the 
time-reversal symmetry and has a finite Chem number n = 
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FIG. 3. Spin and charge densities in orbitally ordered phases of the 
bilayer system as seen from the (111) direction. The bright and dark 
orbitals in the spin nematic and AFM phase denote opposite major- 
ity spin densities and the orbital ordering also breaks the three-fold 
rotation symmetry. The relevant orbitals in the topological phase pre- 
serves the three-fold rotation symmetry. 



±2 (QAH2) or it preserves the time-reversal symmetry but 
breaks the spin-rotation symmetry realizing the quantum spin 
Hall (QSH) state.2 On the mean-field level, QAH and QSH 
phases are degenerate (in fact, for / > J QAH and for / < J 
QSH is favored). The topological phase is surrounded by a 
spin nematic (SN) phase which also develops a weak FM or- 
der (wFM) for increasing U/t. For < 0.9 it is difficult to 
numerically resolve the energy difference between the topo- 
logical and the spin nematic phases because both energies are 
exponentially small in U/t. The phase boundary shown in 
Fig. 2(a) for U/t < 0.9 is an extrapolation to J/U = 0.2 
for U/t which is the result obtained from the analy- 
sis of a reduced model in the next section. Finally, we note 
that charge nematic (CN) phases are absent for the considered 
parameters. 

We now briefly discuss some aspects of perturbing the ideal 
system with Hperp = +Hjt. In Fig. 2(b) we show the re- 
sulting phase diagram for fixed interaction parameters U = 2t 
and J = O.lt which corresponds to the SN phase in the ideal 
model. For finite K and V, the SN phase is accompanied by 
weak /^rr/-magnetic order. Interestingly, because the Jahn- 
Teller interaction favors staggered orbital order, it destabilizes 
the SN phase with uniform orbital order, allowing the topolog- 
ical phase to be energetically favored for some intermediate 
values of K/t and small V/t. For even larger values of K/t, 
we find an AFM phase with staggered orbital order (AFO). 

The mean-field analysis reveals various types of ordering 
of the orbital degrees of freedom. In Fig. 3 we sketch the 
charge and spin density distribution for some representative 
examples. In the SN and the AFM phase the three-fold rota- 
tion symmetry of the lattice is broken and the orbital ordering 
involves real orbitals of the form 



\0) =cos-\d^2) -sin-|42_^2 



(8) 



The AFM phase orders in orbitals with ^afm = (or ±27r/3) 
which are eigenstates of (or the equivalent operators ob- 
tained by rotating T by ±27r/3 around the y axis in orbital 
space). The spin nematic phase is either ordered along the z 
axis (SN^) or the x axis (SN^^) in orbital space and electrons of 
a given spin are predominantly in one of the two orbital eigen- 



states. On the other hand, (T) points along the y direction for 
topological phases and the ordering involves complex orbitals 
of the form given in Eq. (1). As opposed to the real orbitals, 
the charge distribution associated with the complex orbital of 
the form Eq. (1) preserves the trigonal symmetry of the bilayer 
system. We note here that a finite spin-dependent ordering in 
the ^-direction formally enters the mean-field Hamiltonian in 
the same way as the intrinsic spin-orbit coupling would. The 
relation between the topological band properties and the com- 
plex orbitals is further discussed in the next section. 



IV. REDUCED MODEL FOR QBC POINT 

The competition among various weak coupling instabilities 
can be discussed in a reduced model which focuses only on 
the bands participating in the quadratic touching point and to 
momenta within a radius A around the origin in /c-space (k • 
p-expansion). The effective model for the QBC point at the 
Fermi energy is found by expanding Ho{k) to order k'^ and 
eliminating the coupling to the higher bands in the same order 
by use of a canonical transformation. In polar coordinates the 
reduced Hamiltonian takes the form 



4, _ V r^^'^ r 



d<j) 
2^ 



k,<p,(j){k,<p,(j\(E)'Hovh{<t>,k). (9) 



"Horb acts on the two-dimensional orbital space defined by the 
bonding orbitals of the bilayer system given by 

|a) = -^(|a,l) + |a,2)) 

for = with a = a, 6. Horb has the standard form of a 
QBC point with (i-wave symmetry^ 



Horb(0, k) = k^ til^t:, sin(20)T^+t^ cos(20)T^ 



(10) 



Here, / denotes the identity and we have introduced the 

pseudo-spin operator T (with eigenvalues ±1) of the reduced 
orbital space (|a), The parameters in Eq. (10) can be re- 
lated to the hopping t entering the full Bloch matrix Eq. (4): 
tj = tz = —tx = 3t/16. Diagonalizing Eq. (9) yields 
two quadratically dispersing bands with different effective 
masses which touch at k = 0: 61,2 (^) = T^^/ (2^1,2), with 
^1,2 = Tti)]' (Fort^ = tj > the lower band is 

flat and mi 00.) The eigenf unctions of Eq. (10) depend on 
the azimuth and have the simple form 

11)0 = sin (j)\d) -\- cos |2)0 = cos (j)\d) — sin (p\b). 

In other words, the orbital character changes twice when en- 
circling the origin in /c- space. 

Lifting the degeneracy: The degeneracy of the spectrum at 
/c = is lifted by a uniform "orbital field" o^r which couples 

to the orbital pseudo-spin T. A finite ^/-component Oy^a- yields 
a ^-independent coupling between the two bands. Adding 
a term Oy^^Ty to Eq. (10) opens an energy gap throughout 
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fc-space: e±,cr(^) = tik^ i y'^l^^~^^l,a' "^^e resulting 
bands are topologically non-trivial and one finds a finite Chern 
number^^'^"^ 




= -sign(o^,^). (11) 



Here, the limit Oy^^/ (^^A^) has been taken. The unit vec- 
tor n denotes the direction of the resulting field which couples 

to the orbital pseudo-spin T: 

n = [-t,k^ sm{2(l)),0y^^,t,k^ cos(20)] ^ /^Jtlk^^ol^. 

Co- measures the winding number of the vector ft around the 
sphere. From Eq. (1 1) we conclude that a finite orbital field in 
the ^-direction results in a topological phase: If Oy^^ = Oy^^, 
the two electrons occupy the same complex orbital realizing 
a QAH state with a total Chern number n = ^^C^j = ±2 
and Hall conductivity a^y = —ne^/h. If Oy^^ = —Oy^^, time- 
reversal symmetry is preserved and n = 0. However, this 
phase has a non-trivial Z2 -invariant and the QSH state is real- 
ized. 

An orbital field which couples to Tx and/or leads to a (j)- 
dependent coupling between the non-interacting bands. As 
a result, the rotation symmetry of the spectrum in /c-space is 
broken and the QBC point splits into Dirac points with Berry 
phases ±7r. Therefore, a orbital field in the (x, 2:) -plane cor- 
responds to a nematic phase. ^'^ 

Mean-field instabilities: We have analyzed the mean-field 
theory in the reduced space of the model Eq. (9) and have 
found that the order parameters of the nematic and topolog- 
ical phases enter through an orbital field which couples to 

T. The CN and QAH order parameters enter through a spin- 
independent field while the SN and the QSH order parameters 
enter through a spin-dependent field with opposite values for 
t and I spins. In either case, the linearized self-consistency 
equations can be solved and we find that the condensation 
energies depend exponentially on the interaction parameters. 
For the nematic phases (v = SN, CN) we find 

AE^ ^ -2^tlK^/u^ exp {-STit./u^) , (12) 

where 7 ~ 10.8 is a numerical factor. The interaction pa- 
rameters enter through the combinations i^sn = {U — J)/8 
and ucN = {U - 5J)/8 and Eq. (12) holds for > 0. Be- 
cause lisN > ^CN for J > the spin nematic is favored over 
the charge nematic phase. For the topological phases we find 
the following condensation energy: 

AE^ ^ -2tlA'^/u^ exp {-Ant^/u^) . (13) 



The effective interactions for QAH and QSH are equal and 
given by = {U — 3J)/8. There is a factor of two dif- 
ferent in the exponent of Eq. (12) and Eq. (13). This differ- 
ence can be traced back to the angular averaging in momen- 
tum space which for the nematic phases effectively reduces 
the interaction parameter in the exponent. The phase bound- 
ary between the topological and nematic phase is obtained by 
equating the condensation energies Eqs. (12) and (13). In the 
limit U/t 0, it suffices to compare the exponents which 
yields a critical ratio {J/U)^ = 1/5. If J/U < 1/5 the topo- 
logical phase is preferred over the SN phase and if J/U > 1/5 
SN is preferred. For small but finite U/t find a monoton- 
ically decreasing phase boundary consistent with the numeri- 
cal results of the full model presented in Fig. 2(a). 

V. CONCLUSION 

In summary, we have discussed a mechanism for sponta- 
neous quantum Hall states in interacting multi-orbital models 
for a class of transition-metal oxide heterostructures. These 
topological phases are stabilized from purely local interac- 
tions and are accompanied by an orbital ordering of complex 
orbitals. In the weak coupling limit, the topological aspects 
can be understood qualitatively within a reduced model ad- 
dressing the instabilities of the QBC point. Our results sug- 
gest that in the weakly interacting limit topological phases are 
most likely found for small Hund coupling in an inversion 
symmetric bilayer. Notably, we find that weak to interme- 
diate Jahn-Teller interaction can help stabilizing a topological 
phase by suppressing its main competitor, the spin nematic 
phase. Furthermore, the mean-field calculations also suggest 
a topological phase at larger interaction strength within the 
strongly ferromagnetic regime. 

The (111) oxide heterostructures discussed in the present 
work offer a large freedom to design the electronic prop- 
erties by suitable material combinations. Besides the two- 
dimensional spin-orbit^^ or interaction-driven topological in- 
sulators one might also engineer more exotic topological 
phases such as the spin-charge separated QSH* phase^^"^^ or 
fractional quantum Hall states. An important future di- 
rection is to incorporate first-principle calculations to study 
the validity of the tight-binding approximation and to identify 
possible candidate materials. 
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Topological insulators from complex orbital order in transition-metal oxides heterostructures 

Supplemental materials 



I. BAND HAMILTONIAN 



Here, we reproduce the tight-binding Hamiltonian for the Cg orbital degrees of freedom in the (1 1 1) bilayer system which also 
includs the direct overlap between the (i-orbitals and higher-order processes which lead to the second neighbor hopping.^ The 
hopping matrix elements in real space are found from the Slater-Koster energy integrals.^ In momentum space, the hopping on 
the honeycomb lattice takes the form 



The matrix elements are given by^ 



£bk 



£abk 



Ho{k) = 



I ^ak ^abk ^ak ^abk^ 
^abk ^bk ^abk ^bk 
^ak ^abk ^ak ^abk 



(SI) 



abk 



-bk 



£abk £bk . 



£ak = —t - 



lit- 



its) cosC^k^)e~ 



-ts 



^(3t + i5)cos(^fca;)e" 



t' 



4cos(^fc^ 
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Here, the unit of length a is chosen as the nearest-neighbor distance on the honeycomb lattice and set to a = 1. The first Brillouin 
zone is a hexagon with the two inequivalent i^T-points at K± = (±47r/(3>/3), 0). The hopping parameter t parametrizes 
the nearest-neighbor hopping due to the a -hybridization with the oxygen p-states as well as the direct a-bonding between 
neighboring d-orbitals. In terms of the Slater-Koster parameters,^ t ~ {pda)'^ / A + {dda), where A is the energy difference 
between the oxygen p-states and the e^-manifold of the transition-metal ion. The parameter ts ~ {ddS) parameterizes the 
remaining direct nearest-neighbor bonding. For the second-neighbor hopping t\ we have assumed that the most important 
processes are mediated by the bonding between nearest-neighbor oxygens as shown in Fig. SI. 




FIG. SI. Second-neighbor processes parametrized by the amplitude t' shown for hopping in the (x, 2/) -plane. The numbers indicate the relative 
factors from the Slater-Koster energy integrals, (a) The diagonal second-neighbor hopping between — orbitals and (b) between 3z'^ — 
orbitals. (c) The off-diagonal hopping vanishes because the two possible paths cancel. 



II. HARTREE-FOCK APPROXIMATION 



We have used the standard Hartree-Fock approximation for the local four fermion terms: 

(dl^^^dsa^) - {dl^^_^d(3cr2) {d\a3^5a4) 
d^cT2 - ^Li dda^ (4^3 d^cT2 + dda^ ) (4^3 d^cT2 ) ) ' (S2) 

Additionally, we have assumed that possible magnetic order is co-linear and we have therefore only kept terms diagonal in 
spin- space 

{dl^di3a') = Saa'{di^d^^). (S3) 

In other words, we only kept the 2:-components of the spin triplet-order parameters. Let us introduce the following operators 
which are diagonal in orbital space 

^ = ^at + ^ai + ^6t + ^6i, 

7h = na^ - rial + ~ ^K, 

P = na^ ^ rial ~ ~ ^bl, 
S = na^ - rial ~ '^bt + '^bl- 

Similarly, we introduce the following orbital off-diagonal operators 

77 = i (4t^^t + dl^dbi - h.c. 
i = dl^db^-dl^dbi^h.c., 
A = i (4t^^t - di^dbi - h 

The expectation values of the above operators are denoted without hat 
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Local interaction 



With the above introduced notation and the assumption of colhnear magnetic order the local interaction reduces to the follow- 
ing form 

Hioc = ]{U^2U' - J)nfi -\{U^2U' - J)n^ 
4 8 

-|(/7 + J)mm + i(/7 + J)m^ 

--A-U + 2U' - J)pp + li-U + 2U' - J)p^ 
4 o 

-\{U-J)ss+^{U-J)s' 
-^{U'-2J-T)xx+l{U'-2J-T)x' 
_!([;'_ 2J + I)^f, + i(C/' - 2J + 

-\{u'+m+l{u'+i)e 

-^{U'-I)XX+^{U'-I)\\ (S4) 

The QAH phase is characterized by r] ^ and the QSH phase by A 7^ 0. Charge nematic phases are characterized by (p, x) 7^ 
(0, 0) and spin nematic phases by (s, ^) ^ (0, 0). 



Jahn-Teller interaction 



Following Ref. [2], the cooperative Jahn-Teller effect gives rise to an effective interaction between the electrons of the form 

HjT = K^rlr'^ (S5) 



with K > 0. Here, / denotes the direction of the nearest-neighbor bond (z, j) and we have introduced the following local 
pseudo-spin operators: 



T, = COS 



with {ux, UyjUz) = (1,2,3). Explicitly, we obtain 



T-i^ — - It 



rV - 



\/3 



where the orbital pseudo-spin operators are defined as 



T — T 



with a the Pauli matrices. The eigenstates of these operators are the real orbitals of the form 



\0) = cos |C^3^2_^2) + sin |42-2y2). 



(S6) 



(S7) 



(S8) 



(S9) 



In fact, the up pseudo-spin state for the operator r- corresponds to an occupied d^p-r"^ orbital. In terms of the pseudo-spin T, 
the cooperative Jahn-Teller interaction is written as 

K 



TT \ ^ \rriZrriZ i OT^X rj-iX i ^foll^ZrT-iX i rriXrriZ ^ 

^JT - ^ Z^y-i -'-i+x + '^^i -l-i+x + [li li^x + li i^+^J 



zu.c. 

K 



J2 T^T^y + ^T^T^+y - ^ ir^T^y + T^Tl^y) +kJ2 T^T^. 



(SIO) 
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where the summation runs over the unit cells of the honeycomb lattice. In the mean-field treatment, we consider the Hartree 
terms and assume a two-sublattice basis. Then, the expectation values are 

^ 1 fpi, ifi e bottom layer; ^ 1 fxi, if ^ ^ bottom layer; ^^^^^ 

2 lp2, if^ ^ top layer. 2 1 ^2, if ^ ^ top layer. 

The Jahn-Teller interaction becomes 

^JT = -^Yl [^^'1^2 +Pi,2Pl + Xi,lX2 + Xi,2Xl] - Y{PlP2 + X1X2) (S12) 

iu.c. iu.c. 

where the summation is over the unit cells i. This form also shows that the Jahn-Teller interaction favors the staggered compo- 
nents of the real orbitals. 

III. MEAN-FIELD THEORY IN THE REDUCED SPACE 
Projection to the reduced space 

We first want to find a uniform transformation which brings the Bloch matrix Eq. (SI) into block-diagonal form up to order 

U^H,U={^^ ^^^0{k') (S13) 

Here, Hi is the 2x2 Bloch matrix of the lower QBC point and H2 the Bloch matrix of the upper one. The matrix U can be found 
in two steps: U = U1U2. First, introduce the bonding and anti-bonding orbitals of the bilayer system by the transformation 



/o 1 -l^ 
10-10 
010 1 

Vi 1 



(S14) 



which diagonalizes the Bloch matrix Ho{k) for /c = with eigenvalues ^3/2{t-\-ts)-\-3t' and a energy difference A = 3{t-\-ts) 
between upper and lower degeneracy points. However, the two sectors are coupled linearly in k\ 

One can get rid of the linear coupling by introducing the second uniform transformation U2: 



This then leads to the Bloch matrix Hi of the reduced space which defines the model Eq. (9) with parameters 

^' = ^ WtTV) ' ^^ = -^^ = 16 ■ ^^^^^ 



Mean-field interaction in the reduced space 

Writing the local mean-field interaction Eq. (S4) in terms of the bonding and anti-bonding orbitals between top and bottom 
layer and keeping only the /c = components acting on the reduced orbital space we find the following reduced mean-field 
interaction 

^int = -Ey^^y^ ^\k,<l>,a){k,<l>,a\<S>nMF{'j) + E,. (S18) 
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The mean fields act as (spin-dependent) orbital fields and 'Hmf is given by 

^mfM= (p + al) -f (S19) 
where the orbital pseudo-spins of the reduced subspace are defined by 

= \d){b\ + \b){d\, fy =i(\~a){b\-\b){d\), = \~a){a\-\b){b\. (S20) 
The energy constant takes the form 

Ec = ^{pI+pD + ^i~sl + si) + ^(A' + f) (S21) 

and we have introduced fj = py and X = Sy. The effective interactions are given by 

U-3J _ U-bJ _ U-J ^^^^^ 

= Up,y = Us,y = ^ , UqN = Up^x = Up,z = ^ , '^SN = Ug^x = Us,z = ^ • (S22) 

We note that the reduced mean-fields are proportional to the uniform components, i.e., 

Pi ^ Pl,l + Pl,2: Si - + (S23) 

where I = x^y^z and the subscript 1 and 2 refers to bottom and top layer, respectively. The staggered components enter only in 
higher order which is not considered here. Diagonalizing the full Hamiltonian Ho + 'Hint for finite order parameters yields the 
following eigenvalues 



e±,a(A:, 0) = tjk^ ± [t^ sin(20)A:2 + + as^]^ + [t^ cos(20)A:2 + + as^]^ + (77 + aX^. (S24) 



Self-consistency in the reduced space 

The self-consistency equations of the reduced model are given by 

Here, |— )/c,0,a denotes the orbital wave-function of a filled band which in general depends on {k, 0, a). 

Topological phases 

The self-consistency equations for the topological phases (7) 7^ or A 7^ 0) assume the following form 

- t dk , ^ = ^ (S26) 
^ Jo ^k^n4 

where k^ = X/tz for QSH and k'^ = 77/^;^ for QAH. After performing the integration, the order parameter and the condensation 
energy are found: 

kl ^ 2A^ exp and A^^ = — ^ ^ ^ exp . (S27) 

Nematic phases 

Let us first introduce polar coordinates in the orbital (x, plane: 

% =pcos{/3p), Py = psm{/3s), 

Sx = scos{Ps), Sy = ssm{Ps)- (S28) 



10 



It is then easy to show that the energy does not depend on the azimuth in Eqs. (S28) and the angles ^p^g are undetermined in the 
reduced model. (Higher order terms entering the full Hamiltonian do select a definite direction, however). The self-consistency 
equations are more complex because the angular integration is non-trivial: 

1 kdk 1 # k^^kl- 2k^ sin^ (j) _ ^^^^^ 

where kj^ = p/tz for u = CN (charge nematic) and kjy = s/tz for u = SN (spin nematic) and we have introduced 

^^^^ /cork\ 

For kjy ^ the k integration is dominated near k = 0. We can therefore expand in small m: 



+ To — To ~ 9 . 9x9 + regular terms, (S31) 



where the regular terms are of order one in the limit ky ^ {) after k integration. The first three terms, however, are logarithmically 
diverging for small kj, and for /cj, ^ we find 

1 t 

^log-^+c^^, (S32) 
47r kf, Ujj 

where c ~ 0.150 is a numerical constant which arises from the regular terms in the expansion Eq. (S31). It follows that the 
condensation energy for a nematic phase is given by 

where 7 = exp(87rc)/4 ^ 10.8 is a numerical factor. 



Boundary between topological phase and spin nematic 



The boundary between topological phase and spin nematic phase in the limit of small U/ti^ given by equating the condensa- 
tion energies of the reduced model, A£^^ = A£^sn, which can be written in the following form: 



1 



7 



y 



exp 



a(l - by) 



x{l-^y)(l-y) 



X = 



U 



y 



j_ 

Z7' 



a = 



327rt^ 



(S34) 



The solution of the above equation defines the boundary y{x). In the limit x ^ we find y{0) = 1/5. For small but finite x 
we find that y{x) is monotonically decreasing, consistent with the numerical results obtained from the full model presented in 
Fig. 2(a) of the main part. 
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